Portable video cameras are coming into common use. Solid-state imaging devices used in common video cameras include charge-transfer solid-state imaging devices, typified by a CCD (Charge Coupled Device) sensor, and X-Y addressing solid-state imaging devices, typified by a CMOS (Complementary Metal Oxide Semiconductor) sensor.
The CMOS sensor has lower power consumption than the CCD sensor. Further, the CMOS sensor is driven by a single low voltage supply and is easily integrated with a peripheral circuit. Accordingly, incorporating the CMOS sensor into an image processing apparatus such as a video camera is under consideration.
It is, however, difficult to record a high-quality moving image or still image using the CMOS sensor as an imaging device of an image processing apparatus such as a video camera. As one of the reasons, a captured image is distorted by camera shake. For the CCD sensor which has already been used as an imaging device of an image processing apparatus, a single value calculated based on camera shake information obtained in one field or one frame is used as a correction required to execute a process of reducing the effects of camera shake. Since exposure time for all pixels is uniform, an image is not distorted. Therefore, shifts caused by camera shake can be corrected using the single value.
On the other hand, the CMOS sensor captures the image of a subject and processes the captured image according to the following mechanism. The image is distorted by camera shake. The distortion may be caused by the following reason.
In the charge-transfer solid-state imaging device such as the CCD sensor, all pixels are simultaneously exposed, so that pixel data can be read out. In the X-Y addressing solid-state imaging device such as the CMOS sensor, data is read out every pixel or data is sequentially read out every line. In the solid-state imaging device, when data is sequentially read out every pixel, the difference in read time between pixels in one line is negligibly smaller than that between lines. In the following description, as shown in FIG. 1, the difference in read time between lines will be considered.
For example, a CMOS sensor will now be described. The sensor has a structure in which, as shown in FIG. 1, read time for all pixels is 1/60 seconds and each frame includes lines 1 to N. Referring to FIG. 2, time t1 is the start of exposing the line 1 and time t2 is the end thereof. Time t3 is the end of exposing the line 2. Time t4 is the end of exposing the line 3.
The difference between time t2 and time t3 corresponds to time difference Δt. The difference between time t3 and time t4 also corresponds to the time difference Δt. In other words, the time difference Δt occurs every line with respect to exposure time. In this case, therefore, the time difference between the top line 1 and the bottom line N in one frame is approximately 1/60 seconds corresponding to the exposure time. When camera shake is given under condition that the exposure time of the bottom line N is delayed from that of the top line 1 in reading one frame, unfortunately, the image of a subject is distorted due to the differences in exposure time between lines.
The problem in that the captured image of a subject is distorted will now be described with reference to FIGS. 3 and 4. FIGS. 3 and 4 show examples of a case where a stationary rectangle is captured as a subject in the center of a frame. The middle diagram of FIG. 3 shows a normal image without being distorted, the image being obtained by shooting the subject through an image processing apparatus without camera shake. The subject is captured as a rectangle.
The right diagram of FIG. 3 shows an image distorted by rightward movement of the image processing apparatus in shooting the subject. Similarly, the left diagram of FIG. 3 shows an image distorted by leftward movement of the image processing apparatus in shooting the subject.
The middle diagram of FIG. 4 shows a normal image without being distorted, the image being obtained by shooting the subject through the image processing apparatus without camera shake. The subject is captured as a rectangle. The upper diagram of FIG. 4 shows an image extended vertically by upward movement of the image processing apparatus in shooting the subject. The lower diagram of FIG. 4 shows an image contracted vertically by downward movement of the image processing apparatus in shooting the subject.
As mentioned above, in the CMOS sensor, exposure time is shifted (i.e., imaging timing is delayed) every line. If shifts caused by camera shake are corrected using a single value calculated from camera shake information obtained in one field or one frame, the effects of camera shake cannot completely be eliminated. Unfortunately, an image to be given to a user may be distorted.
To correct image distortion caused by camera shake, an image processing apparatus for changing a read position every line in order to correct shifts in the horizontal direction and changing the position of a line to be read in the vertical direction in order to correct shifts in the vertical direction is proposed (refer to, e.g., Japanese Unexamined Patent Application Publication No. 2001-358999).
According to a method disclosed in Japanese Unexamined Patent Application Publication No. 2001-358999, it is assumed that camera shake information is obtained every line constituting a frame. Disadvantageously, it is difficult to obtain camera shake information every line in terms of the sampling frequency of a sensor for detecting camera shake.
In addition, according to the method disclosed in Japanese Unexamined Patent Application Publication No. 2001-358999, unfortunately, shifts caused by camera shake are corrected in units of pixels.
It is necessary for conventional image processing apparatuses to temporarily store image data of a captured image. Accordingly, the conventional image processing apparatus requires a memory having a capacity capable of storing data of at least one frame. Therefore, the memory capacity cannot be reduced.